Get Tables List
Retrieve the list of existing tables in the database with their record counts.
Get Tables List
Processing
Retrieves the list of existing tables in the database with their record counts. The function connects to various database types (PostgreSQL, MySQL, MariaDB, Oracle Database, Microsoft SQL Server, SQLite) and returns information about all tables including their names and row counts.
Inputs
- database name
- Name of the database to connect to. For SQLite, this is the path to the database file.
- host (optional)
- Database server hostname or IP address. Not used for SQLite databases.
- port (optional)
- Database server port number. Must be between 1 and 65535. Not used for SQLite databases.
- username (optional)
- Database username for authentication. Not used for SQLite databases.
- password (optional)
- Database password for authentication. Supports SecretStr for secure handling. Not used for SQLite databases.
⚠️ The function supports environment variable configuration as an alternative to direct input parameters. Input parameters are checked first; if not available, the corresponding environment variable is used.
Inputs Types
Input | Types |
---|---|
database name |
Str |
host |
Str |
port |
Int |
username |
Str |
password |
SecretStr |
You can check the list of supported types here: Available Type Hints.
Outputs
- tables data
- List of database tables with their metadata including table names and row counts. Format depends on the selected output type option.
The tables data
output contains the following metadata for each table:
- table_name: Name of the database table
- row_count: Number of records/rows in the table
Outputs Types
Output | Types |
---|---|
tables data |
DataFrame , DataRecords |
You can check the list of supported types here: Available Type Hints.
Options
The Get Tables List brick contains some changeable options:
- Database type
- Specifies which database system to connect to. Supports PostgreSQL, MySQL, MariaDB, Oracle Database, Microsoft SQL Server, and SQLite.
- Output type
- Determines the format of the returned data. Can be Records (Python list), Pandas DataFrame, or Polars DataFrame.
- Env variables prefix
- Optional prefix for environment variable names. When specified, environment variables will be read as
{prefix}_{VARIABLE_NAME}
instead of just{VARIABLE_NAME}
. - Verbose
- Controls logging output. When enabled, provides detailed information about connection process and any errors encountered.
Environment Variables Configuration
If you don't provide database connection parameters as brick inputs, you can configure them using environment variables.
Step 1 — Create an environment file
Create an environment file in the flow project folder. You can name it anything you like (e.g. .env
, db.env
, or mysettings.env
).
Example: db.env
# Example database environment configuration
DB_NAME=mydatabase
DB_HOST=localhost
DB_PORT=5432
DB_USERNAME=myuser
DB_PASSWORD=mypassword
💡 For SQLite, only
DB_NAME
is required (e.g.,DB_NAME=mydb.sqlite
).
Step 2 — Reference the env file in CODED FLOWS
Inside the CODED FLOWS app, use the LOAD ENV control brick to load the file you created.
- Specify your env filename (
db.env
for example or whichever name you chose). - All defined variables become available to other any bricks in the connected graph flow.
Step 3 — Use with prefix (optional)
If you enable the "Env variables prefix" option in the LOAD ENV brick, the system will automatically prepend the prefix and an underscore to each variable.
Example with prefix MYAPP
:
MYAPP_DB_NAME=mydatabase
MYAPP_DB_HOST=localhost
MYAPP_DB_PORT=5432
MYAPP_DB_USERNAME=myuser
MYAPP_DB_PASSWORD=mypassword
Bricks will then look for MYAPP_DB_NAME
, MYAPP_DB_HOST
, etc. instead of the default variable names.
import logging
import sqlalchemy as sa
from sqlalchemy import inspect, func, select, Table, MetaData
import pandas as pd
import polars as pl
from os import getenv
from coded_flows.types import Str, SecretStr, Int, DataFrame, DataRecords, Union
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def _coalesce(*values):
return next((v for v in values if v is not None))
def get_tables_list(
database_name: Str = None,
host: Str = None,
port: Int = None,
username: Str = None,
password: SecretStr = None,
options=None,
) -> Union[DataFrame, DataRecords]:
brick_display_name = "Get Tables List"
tables_data = None
options = options or {}
verbose = options.get("verbose", True)
db_type = options.get("db_type", "SQLite")
out_type = options.get("out_type", "Pandas")
env_prefix = options.get("env_prefix", "")
env_prefix = env_prefix + "_" if env_prefix.strip() else ""
try:
database_name = _coalesce(database_name, getenv(f"{env_prefix}DB_NAME"))
if db_type != "SQLite":
host = _coalesce(host, getenv(f"{env_prefix}DB_HOST"))
port = _coalesce(port, getenv(f"{env_prefix}DB_PORT"))
username = _coalesce(username, getenv(f"{env_prefix}DB_USERNAME"))
password = _coalesce(password, getenv(f"{env_prefix}DB_PASSWORD"))
if not database_name or (
db_type != "SQLite"
and (not host or not port or (not username) or (not password))
):
verbose and logger.error(
f"[{brick_display_name}] Either the env file is not defined, not referenced in the project folder, or one or more DB configuration inputs are missing."
)
raise ValueError(
"Either the env file is not defined, not referenced in the project folder, or one or more DB configuration inputs are missing."
)
try:
if hasattr(password, "get_secret_value"):
password = password.get_secret_value()
else:
password = str(password)
except Exception:
password = str(password)
except Exception as e:
verbose and logger.error(
f"[{brick_display_name}] Either the env file is not defined, not referenced in the project folder, or one or more DB configuration inputs are missing."
)
raise ValueError(
"Either the env file is not defined, not referenced in the project folder, or one or more DB configuration inputs are missing."
)
try:
if port:
port = int(port)
except (ValueError, TypeError):
verbose and logger.error(
f"[{brick_display_name}] Port must be an integer, got {port!r}"
)
raise ValueError(f"Port must be an integer, got {port!r}")
if port and (not 1 <= port <= 65535):
verbose and logger.error(
f"[{brick_display_name}] Port must be between 1 and 65535, got {port}"
)
raise ValueError(f"Port must be between 1 and 65535, got {port}")
verbose and logger.info(
f"[{brick_display_name}] Retrieving list of tables from {db_type} database."
)
driver_map = {
"Postgres": "postgresql+psycopg",
"MySQL": "mysql+pymysql",
"MariaDB": "mariadb+pymysql",
"Oracle Database": "oracle+oracledb",
"Microsoft SQL Server": "mssql+pymssql",
"SQLite": "sqlite",
}
if db_type == "SQLite":
if host or username or password:
verbose and logger.warning(
f"[{brick_display_name}] Ignoring host, username, password for SQLite."
)
url = f"{driver_map[db_type]}:///{database_name}"
else:
if not all([host, str(port), username, password, database_name]):
verbose and logger.error(
f"[{brick_display_name}] Missing required connection parameters for {db_type}."
)
raise ValueError(
f"[{brick_display_name}] Missing required connection parameters for {db_type}."
)
url = f"{driver_map[db_type]}://{username}:{password}@{host}:{port}/{database_name}"
verbose and logger.info(f"[{brick_display_name}] Connection URL constructed.")
try:
engine = sa.create_engine(url)
verbose and logger.info(f"[{brick_display_name}] Engine created.")
inspector = inspect(engine)
table_names = inspector.get_table_names()
data = []
metadata = MetaData()
with engine.connect() as connection:
for table in table_names:
table_obj = Table(table, metadata)
count_query = select(func.count()).select_from(table_obj)
row_count = connection.scalar(count_query)
data.append({"table_name": table, "row_count": row_count})
if out_type.lower() == "pandas":
tables_data = pd.DataFrame(data)
elif out_type.lower() == "polars":
tables_data = pl.DataFrame(data)
else:
tables_data = data
engine.dispose()
verbose and logger.info(
f"[{brick_display_name}] Tables list retrieved successfully. Tables found: {(len(tables_data) if tables_data is not None else 0)}"
)
except Exception as e:
verbose and logger.error(
f"[{brick_display_name}] An error occurred while retrieving tables list: {str(e)}"
)
raise
return tables_data
Brick Info
- cryptography
- sqlalchemy
- psycopg[binary]
- pymysql
- pymssql
- pandas
- polars
- oracledb