Merge pull request #57 from stackql/feature/refactor

v3.8.0

Author: jeffreyaven

CHANGELOG.md

@@ -1,9 +1,10 @@
 # Changelog
 
-## v3.8.0 (2025-06-04)
+## v3.8.0 (2025-06-13)
 
 ### Updates
 
+- Added `--csv-download` argument for stackql magic commands
 - Refactor
 - Enhanced test coverage
 

README.rst

@@ -22,7 +22,7 @@ PyStackQL - Python Wrapper for StackQL
 StackQL is an open source developer tool which allows you to query and interact with cloud and SaaS provider APIs using SQL grammar.
 StackQL can be used for cloud inventory analysis, cloud cost optimization, cloud security and compliance, provisioning/IaC, assurance, XOps, and more.
 
-PyStackQL is a Python wrapper for StackQL which allows you to use StackQL within Python applications and to use the power of Python to extend StackQL.
+`PyStackQL <https://pypi.org/project/pystackql/>`_ is a Python wrapper for StackQL which allows you to use StackQL within Python applications and to use the power of Python to extend StackQL.  
 PyStackQL can be used with ``pandas``, ``matplotlib``, ``plotly``, ``jupyter`` and other Python libraries to create powerful data analysis and visualization applications.
 
 For detailed documentation, including the API reference, see `Read the Docs <https://pystackql.readthedocs.io>`_.
@@ -52,19 +52,17 @@ The following example demonstrates how to run a query and return the results as
 ::
 
     from pystackql import StackQL
-    import pandas as pd
     region = "ap-southeast-2"
-    stackql = StackQL()
+    stackql = StackQL(output='pandas')
     
     query = """
-    SELECT instanceType, COUNT(*) as num_instances
+    SELECT instance_type, COUNT(*) as num_instances
     FROM aws.ec2.instances
     WHERE region = '%s'
-    GROUP BY instanceType
+    GROUP BY instance_type
     """ % (region)   
     
-    res = stackql.execute(query)
-    df = pd.read_json(res)
+    df = stackql.execute(query)
     print(df)
 
 Using PyStackQL with Jupyter Notebook
@@ -76,7 +74,7 @@ To use the integrated Jupyter magic commands provided by PyStackQL:
 
 .. code-block:: python
 
-    %load_ext pystackql
+    %load_ext pystackql.magic
 
 2. **Execute a Query Using Line Magic**:
 
@@ -111,12 +109,11 @@ Supported Python Versions
 
 PyStackQL has been tested on:
 
-- Python 3.7
-- Python 3.8
 - Python 3.9
 - Python 3.10
 - Python 3.11
-- Python 3.12 (MacOS and Linux only)
+- Python 3.12
+- Python 3.13
 
 Licensing
 ~~~~~~~~~

docs/source/conf.py

@@ -20,13 +20,13 @@
 # -- Project information -----------------------------------------------------
 
 project = 'pystackql'
-copyright = '2021-2024, StackQL Studios'
+copyright = '2021-2025, StackQL Studios'
 author = 'StackQL Studios'
 
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = 'v3.6.6'
+release = 'v3.8.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/source/examples.rst

@@ -49,10 +49,10 @@ StackQL can be used to collect, analyze, summarize, and report on cloud resource
     regions = ["ap-southeast-2", "us-east-1"]
     queries = [
         f"""
-        SELECT '{region}' as region, instanceType, COUNT(*) as num_instances
+        SELECT '{region}' as region, instance_type, COUNT(*) as num_instances
         FROM aws.ec2.instances
         WHERE region = '{region}'
-        GROUP BY instanceType
+        GROUP BY instance_type
         """
         for region in regions
     ]

docs/source/intro.rst

@@ -71,19 +71,17 @@ The following example demonstrates how to run a query and return the results as
 .. code-block:: python
 
     from pystackql import StackQL
-    import pandas as pd
     region = "ap-southeast-2"
-    stackql = StackQL()
+    stackql = StackQL(output='pandas')
     
     query = """
-    SELECT instanceType, COUNT(*) as num_instances
+    SELECT instance_type, COUNT(*) as num_instances
     FROM aws.ec2.instances
     WHERE region = '%s'
-    GROUP BY instanceType
+    GROUP BY instance_type
     """ % (region)   
     
-    res = stackql.execute(query)
-    df = pd.read_json(res)
+    df = stackql.execute(query)
     print(df)
 
 Using ``UNION`` and ``JOIN`` operators
@@ -96,26 +94,25 @@ StackQL is a fully functional SQL programming environment, enabling the full set
     ...
     regions = ["ap-southeast-2", "us-east-1"]
     query = """
-    SELECT '%s' as region, instanceType, COUNT(*) as num_instances
+    SELECT '%s' as region, instance_type, COUNT(*) as num_instances
     FROM aws.ec2.instances
     WHERE region = '%s'
-    GROUP BY instanceType
+    GROUP BY instance_type
     UNION
-    SELECT  '%s' as region, instanceType, COUNT(*) as num_instances
+    SELECT  '%s' as region, instance_type, COUNT(*) as num_instances
     FROM aws.ec2.instances
     WHERE region = '%s'
-    GROUP BY instanceType
+    GROUP BY instance_type
     """ % (regions[0], regions[0], regions[1], regions[1])
     
-    res = stackql.execute(query)
-    df = pd.read_json(res)
+    df = stackql.execute(query)
     print(df)
 
 The preceding example will print a ``pandas.DataFrame`` which would look like this:
 
 .. code-block:: sh
 
-      instanceType  num_instances          region
+     instance_type  num_instances          region
     0    t2.medium              2  ap-southeast-2
     1     t2.micro              7  ap-southeast-2
     2     t2.small              4  ap-southeast-2
@@ -133,17 +130,15 @@ In addition to ``UNION`` DML operators, you can also run a batch (list) of queri
 
     queries = [
         f"""
-        SELECT '{region}' as region, instanceType, COUNT(*) as num_instances
+        SELECT '{region}' as region, instance_type, COUNT(*) as num_instances
         FROM aws.ec2.instances
         WHERE region = '{region}'
-        GROUP BY instanceType
+        GROUP BY instance_type
         """
         for region in regions
     ]
 
-    res = stackql.executeQueriesAsync(queries)
-    df = pd.read_json(json.dumps(res))
-
+    df = stackql.executeQueriesAsync(queries)
     print(df)
 
 
@@ -156,10 +151,9 @@ Here is an example of using the ``json_extract`` function to extract a field fro
 .. code-block:: python
 
     from pystackql import StackQL
-    import pandas as pd
     subscriptionId = "273769f6-545f-45b2-8ab8-2f14ec5768dc"
     resourceGroupName = "stackql-ops-cicd-dev-01"
-    stackql = StackQL()
+    stackql = StackQL() # output format defaults to 'dict'
 
     query = """
     SELECT name,  
@@ -172,8 +166,7 @@ Here is an example of using the ``json_extract`` function to extract a field fro
     """ % (resourceGroupName, subscriptionId)
     
     res = stackql.execute(query)
-    df = pd.read_json(res)
-    print(df)
+    print(res)
 
 Using the Jupyter Magic Extension
 =================================
@@ -184,7 +177,7 @@ To get started with the magic extension, first load it into your Jupyter environ
 
 .. code-block:: ipython
 
-    %load_ext pystackql
+    %load_ext pystackql.magic
 
 After loading the magic extension, you can use the `%%stackql` magic to execute StackQL commands in a dedicated Jupyter cell. The output will be displayed directly below the cell, just like any other Jupyter command output.
 
@@ -196,3 +189,9 @@ Example:
     SHOW SERVICES in aws
 
 This Jupyter magic extension provides a seamless integration of `pystackql` into your Jupyter workflows, allowing you to explore cloud and SaaS provider data interactively within your notebooks.
+
+To use the magic extension to run queries against a StackQL server, you can use the following command:
+
+.. code-block:: ipython
+
+    %load_ext pystackql.magics

docs/source/magic_ext.rst

@@ -38,25 +38,39 @@ The extension provides both line and cell magic functionalities:
    .. code-block:: python
 
         %%stackql
-        SELECT instanceType, COUNT(*) as num_instances
+        SELECT instance_type, COUNT(*) as num_instances
         FROM aws.ec2.instances 
-        WHERE region = '$region' GROUP BY instanceType       
+        WHERE region = '$region' GROUP BY instance_type       
 
 Options
 -------
 
 When using `StackqlMagic` as cell magic, you can pass in the following options:
 
 - ``--no-display`` : Suppresses the display of the results. Even when this option is enabled, the results are still saved in the `stackql_df` Pandas DataFrame.
+- ``--csv-download`` : Adds a download button below the query results that allows you to download the data as a CSV file.
 
-Example:
+Examples
+--------
+
+Basic Query
+~~~~~~~~~~
 
 .. code-block:: python
 
     project = 'stackql-demo'
     zone = 'australia-southeast1-a'
     region = 'australia-southeast1'
 
+    %%stackql
+    SELECT SPLIT_PART(machineType, '/', -1) as machine_type, count(*) as num_instances 
+    FROM google.compute.instances 
+    WHERE project = '$project' AND zone = '$zone'
+    GROUP BY machine_type
+
+Suppressing Display
+~~~~~~~~~~~~~~~~~~
+
 .. code-block:: python
 
     %%stackql --no-display
@@ -67,6 +81,38 @@ Example:
 
 This will run the query but won't display the results in the notebook. Instead, you can later access the results via the `stackql_df` variable.
 
+Downloading Results as CSV
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    %%stackql --csv-download
+    SELECT 
+        'Python' as language,
+        'Development' as mode,
+        'PyStackQL' as package
+
+This will display the query results in the notebook and add a download button below the results. Clicking the button will download the data as a CSV file named ``stackql_results.csv``.
+
+Combining Options
+~~~~~~~~~~~~~~~
+
+You can also combine options. For example, if you want to suppress the display but still want a download button:
+
+.. code-block:: python
+
+    # First run the query with no display
+    %%stackql --no-display
+    SELECT instance_type, COUNT(*) as num_instances
+    FROM aws.ec2.instances 
+    WHERE region = '$region' GROUP BY instance_type
+
+    # Then manually display with the download button
+    from IPython.display import display
+    display(stackql_df)
+    from pystackql import StackqlMagic
+    StackqlMagic(get_ipython())._display_with_csv_download(stackql_df)
+
 .. note::
 
     The results of the queries are always saved in a Pandas DataFrame named `stackql_df` in the notebook's current namespace. This allows you to further process or visualize the data as needed.
@@ -75,8 +121,8 @@ An example of visualizing the results using Pandas is shown below:
 
 .. code-block:: python
 
-    stackql_df.plot(kind='pie', y='num_instances', labels=_['machine_type'], title='Instances by Type', autopct='%1.1f%%')
+    stackql_df.plot(kind='pie', y='num_instances', labels=stackql_df['machine_type'], title='Instances by Type', autopct='%1.1f%%')
 
 --------
 
-This documentation provides a basic overview and usage guide for the `StackqlMagic` extension. For advanced usage or any additional features provided by the extension, refer to the source code or any other accompanying documentation.
+This documentation provides a basic overview and usage guide for the `StackqlMagic` extension. For advanced usage or any additional features provided by the extension, refer to the source code or any other accompanying documentation.