There has never been a greater demand for modern data platforms than we witness right now. Data-driven businesses are increasingly conscious of the power of data. With the introduction of Snowflake Snowpark, they now have the opportunity to reap even greater benefits from it.
Snowflake Snowpark is a true game-changer for scientists, engineers, and analysts, now allowing them to seamlessly collaborate using their preferred programming languages and tools, all within a single, unified platform. It’s an indispensable tool for data management, analytics, and machine learning. The best part? It comes at no extra cost when you’re already using Snowflake.
In this blog post, I’m diving into the essence of Snowpark, exploring its features and benefits. Let’s discover how this great tool can help you unlock new possibilities, streamline workflows, and accelerate innovation within your organization right now.
What is Snowflake Snowpark?
Snowflake Snowpark is a developer framework designed to enhance and simplify the way data professionals interact with the Snowflake. It allows users to bring their code to the data rather than moving data to the code, enabling more efficient data processing, transformation, and analysis – all of this directly within Snowflake's platform.
Take a look at these key features of Snowflake Snowpark:
- Support for multiple programming languages: initially launched with support for Scala, Snowpark has expanded to include support for other programming languages such as Java and Python. This makes it accessible to a broader range of developers and data scientists, who can now use familiar languages to interact with data in Snowflake.
- DataFrame API: Snowpark provides a DataFrame API that enables complex data manipulations and transformations using a familiar programming model. This API abstracts away the underlying SQL, making it easier for users to perform sophisticated data processing tasks without deep SQL expertise.
- User-Defined Functions (UDFs) and stored procedures: it supports the creation of custom user-defined functions and stored procedures in supported languages. This allows for custom logic and algorithms to be executed directly within the Snowflake environment, improving performance and security.
- Secure and governed data sharing: leveraging Snowflake’s robust data sharing capabilities, Snowpark allows users to securely share and access shared data without the need to copy or move it. This ensures data governance and privacy are maintained.
- Integration with ML libraries: Snowpark enables the integration with popular machine learning libraries and frameworks, allowing data scientists to build and train models directly within Snowflake. This streamlined workflow can significantly speed up the development and deployment of machine learning models.
How does Snowflake Snowpark work?
In a nutshell, Snowpark lets your developers build data pipelines, machine learning models, and data applications directly within Snowflake. This not only keeps things streamlined and secure, but also taps into the full power of Snowflake's processing capabilities.
Snowpark works on two levels: the client side and the server side. Let’s take a closer look at each.
Client side
Libraries
Snowpark provides libraries compatible with Python, Java, or Scala runtimes, available for integration in various environments, including local development setups, notebooks, or production environments for deployment automation. These libraries facilitate the importation of essential functions and classes for working with Snowpark. Python users have the added advantage of directly integrating Snowpark Python code into Python worksheets in Snowsight.
The libraries contain functions for establishing connections with Snowflake. For instance, in Python, initiating a new session within Snowflake involves creating a dictionary with connection parameters, using this dictionary with the <Session.builder.configs> method to generate a builder object, and then calling the builder's <create> method to start the session.
DataFrames
DataFrames are central to querying and processing data in Snowpark, acting as relational datasets that are evaluated lazily. They execute only when triggered by a specific action. DataFrames can be generated from various sources, including tables, views, streams, stages, SQL query results, or hard-coded values.
Transformations
DataFrames allow the application of various operations, such as filtering rows, selecting columns, transforming values (including custom user-defined functions (UDFs)), aggregating data, and joining with other DataFrames. Transformations return a new DataFrame object without altering the original, supporting method chaining for multiple transformations.
Evaluation actions
The lazy evaluation of DataFrames means the SQL statement is executed on the server only when an action triggers it. Actions like <collect>, <count>, <show>, or saving the DataFrame as a table prompt this evaluation.
UDFs and stored procedures
Snowpark allows the creation and use of UDFs and stored procedures for data processing within your DataFrame. When a UDF is created via the Snowpark API, the function's code is uploaded to an internal stage in Snowflake, ensuring data does not need to move to the client for processing.
Server side
Execution plan
Snowpark operations send optimized SQL code and instructions to Snowflake servers. An execution plan is formulated server-side, mapping logical operations to Snowflake's data processing mechanisms, taking advantage of its architecture for parallel data access and processing across multiple nodes.
Resource management
Snowflake dynamically allocates necessary resources (CPU, memory, and bandwidth) for tasks, automatically scaling to manage large workloads efficiently.
Results
Upon task completion, results are compiled into a DataFrame and sent back to the client-side library for further action or display.
Container services
Snowpark Container Services offer a fully managed container environment within the Snowflake ecosystem, facilitating the deployment, management, and scaling of containerized applications. It provides an OCI runtime execution environment optimized for Snowflake, allowing seamless execution of OCI images without moving data out of Snowflake.
This feature supports the deployment of specialized analytics, machine learning models, or any custom application as a container, ensuring seamless execution and integration with Snowflake's data storage and processing capabilities.
Snowflake Snowpark advantages for modern data platforms
Snowpark introduces a suite of advantages to modern data platforms and data-driven companies, fundamentally enhancing how they handle, analyze, and derive value from their data. Here are the key benefits:
Unified environment
Snowpark enables developers, data scientists, and engineers to work within a single, integrated environment, significantly streamlining the data processing workflows. It reduces the complexity associated with using separate systems for different types of data tasks.
Efficient data operations
Snowpark's approach of executing code directly on the data stored in Snowflake reduces data movement, which not only enhances security but also decreases the time and costs associated with transferring large data sets across systems.
In addition, by leveraging Snowflake's automatic scaling and efficient query execution, Snowpark ensures high performance for data operations, handling spikes in demand without manual intervention.
Simplified data engineering
Snowpark enables data engineers to write and execute complex data transformation pipelines directly within the Snowflake environment using familiar programming languages like Java, Scala, and Python.
This integration significantly simplifies the data engineering process by removing the need to manage separate data processing systems, thereby streamlining the development and deployment of scalable data pipelines.
Advanced analytics and ML
Snowpark facilitates the integration of machine learning frameworks, enabling the development and execution of models directly within Snowflake. In fact, in conjunction with Snowflake's machine learning capabilities, Snowpark offers an end-to-end solution for developing, training, and deploying machine learning models entirely within the data platform. This includes everything from data preprocessing and feature engineering to model training and inference, streamlining the machine learning lifecycle, and enabling rapid deployment of predictive models.
On top of that, the ability to create UDFs and leverage external libraries within Snowpark expands the possibilities for custom analytics solutions tailored to specific business needs.
Enhanced security and data governance
Snowpark benefits from Snowflake's robust security and governance features, including data encryption, access controls, and audit trails, ensuring that data processing and analysis are secure and compliant with regulations.
The unified platform approach also helps maintain clear data lineage and governance, simplifying compliance and data management efforts.
Cost-effective scalability
Snowflake's serverless architecture and automatic scaling reduce the need for manual resource provisioning and optimization, ensuring cost-effective scalability. Companies pay only for the compute resources they use, optimizing their investment.
Snowpark can also lead to lower infrastructure complexity and costs, as it minimizes the need for external data processing and analytics tools.
Interoperability and flexibility
Snowpark's support for multiple programming languages enhances its interoperability within the broader technology ecosystem. This flexibility enables organizations to leverage existing skills and tools, facilitating a smoother integration of Snowpark into their data platforms.
Moreover, Snowpark's DataFrame API provides a familiar interface for data manipulation, making it accessible to a wide range of users from different technical backgrounds.
No additional cost
Indeed – Snowpark is included in a Snowflake subscription, so if your data platform is built on Snowflake, you can benefit from Snowpark completely free of charge!
What are the use cases of Snowpark?
Snowpark introduces a suite of powerful capabilities designed to further elevate the capabilities of modern data platforms. It bridges the gap between complex data engineering, data science tasks, and the scalable, secure environment of Snowflake. You can leverage Snowpark in the following ways:
- Data transformation and preprocessing: Snowpark can be used to perform complex data transformations, such as joining multiple tables, filtering data, and aggregating results. You can also Leverage Snowpark's DataFrame API to manipulate and preprocess data before feeding it into machine learning models.
- Machine learning and data science applications: you can develop and train ML models directly within Snowflake using Snowpark, using machine learning, popular libraries like scikit-learn, XGBoost, or TensorFlow with Snowpark to build and deploy them seamlessly. Feature engineering, model training and scoring can also be done within Snowflake, eliminating the need to move data outside the platform.
- Data pipeline and ELT workloads: construct end-to-end data pipelines using Snowpark, combining data ingestion, loading and processes.. Further integrating Snowpark with other tools and frameworks like Apache Spark or Airflow will allow you to build robust and scalable data pipelines.
- Data validation and quality checks: Snowpark can help you swiftly implement data validation rules and quality checks to ensure data integrity and consistency. It also allows you to write custom validation logic to identify and handle data anomalies or inconsistencies.
- Data enrichment and augmentation: you can enhance existing datasets by joining them with external data sources or APIs using Snowpark. Then, perform data enrichment tasks, such as geocoding, sentiment analysis, or entity extraction, directly within Snowflake.
- Ad-hoc data analysis and exploration: you can enable quick data insights and visualizations with Snowpark's interactive features (such as Jupyter notebooks).
- Custom analytics and reporting: combine SQL and procedural programming to develop custom analytics and reporting solutions using Snowpark – you can generate complex reports, dashboards, or data visualizations by processing and aggregating data as you please
- Data security and access control: Snowpark allows you to implement fine-grained access control and data security measures. Do you need custom authentication and authorization logic to ensure data privacy and compliance with regulatory requirements? You can write those easily as well.
These are just a few examples of how Snowflake Snowpark can help businesses streamline data processing, analytics, and ML workflows. Snowpark's flexibility and integration with popular programming languages make it a versatile tool for a wide range of data-related tasks within the Snowflake ecosystem.
Case study 1: Pfizer
Pfizer's migration to Snowflake and the utilization of its features like Snowpark and Snowgrid enabled the company to unify data, improve accessibility, speed up data processing, enhance collaboration, strengthen data governance, and significantly reduce costs. These factors collectively contributed to Pfizer lowering TCO by 57% and achieving a 4x faster data processing speed.
Client: Pfizer, a global pharmaceutical company with 80,000 employees across six continents.
Challenge:
- Data silos due to disparate systems from mergers and acquisitions
- Difficulty in timely access to data for critical decision-making
- Costly delays due to constant data extraction, movement, and transformation
Solution:
- Adopted the Snowflake data platform to unify data into a single source of truth
- Utilized Snowpark to create a cloud-based Virtual Analytics Workspace for collaboration
- Leveraged Snowgrid for seamless and secure cross-region, cross-cloud data sharing
- Implemented Snowflake Secure Data Sharing for faster data integration during M&A transitions
Results:
- 19,000 annual hours saved through faster data processing
- 57% reduction in TCO and 28% reduction in overall database costs compared to the previous solution
- Accelerated sales reports and up to 4x faster analytics
- Improved business continuity and supply chain visibility
- Enhanced data governance and security across all business domains
- Seamless data sharing and integration during M&A transitions
- Management views Snowflake as key to Pfizer's strategic IT initiatives
Read the full case study.
Case study 2: Sanofi
Sanofi's migration to Snowflake and the adoption of Snowpark enabled the company to redesign its analytical engine, streamline data processing, and improve performance by 50% compared to their previous managed Spark solution. By leveraging Snowflake's separation of storage and compute, near-zero maintenance, and on-demand scalability, Sanofi processed 100 million patient records per cohort in an average of four minutes. The migration also enhanced data governance, reduced latency, and lowered overall TCO.
Client: Sanofi, a global healthcare company with more than 100,000 employees worldwide, including 13,000 professionals in the United States.
Challenge:
- Developing an enterprise-wide data processing platform to assist the medical community in analyzing real-world clinical data for drug discovery
- Challenges with the managed Spark solution, including manual deployment and maintenance, resource scalability issues, pipeline failures, concurrency problems, and data movement complexities
Solution:
- Redesigned analytical engine using Snowflake and Snowpark
- Leveraged Snowflake's separation of storage and compute, near-zero maintenance, and on-demand scalability
- Implemented a service-focused architecture to enhance fault isolation and minimize disruption during migration
- Utilized Snowflake's features for data governance, including granular permissions and role-based access control
Results:
- 50% improvement in performance compared to the previous managed Spark solution
- Processed 100 million patient records per cohort in an average of four minutes
- Reduced latency and improved overall performance, enabling faster data processing and analytics
- Ensured data security and compliance with policies
- Lowered overall TCO and reduced data movement costs
- Engaged with Snowflake Professional Services for a successful migration, utilizing the Snowpark Migration Accelerator to convert PySpark code to Snowpark
Read the full case study.
Case study 3: Vanderbilt University
Vanderbilt University's adoption of Snowflake's Data Cloud enabled the institution to unify data, streamline analytics, and enhance data governance. By ingesting constituent data from Salesforce and leveraging Snowflake Marketplace, Vanderbilt empowered its development and alumni relations teams with timely insights, leading to improved donor engagement and optimized advancement efforts. Additionally, Vanderbilt achieved a 4x improvement in Salesforce ETL throughput by utilizing AWS AppFlow and Snowpark for Python, reducing load times and vendor dependency.
Client: Vanderbilt University, a private research university in Nashville, Tennessee, with 13,710 enrolled students and over 150,000 living alumni.
Challenge:
- Efficiently managing resources while ensuring a stellar student experience and conducting transformational research
- Difficulty in unifying and leveraging data due to the on-premises data environment
- Data wrangling required by technical staff to answer questions
- Data governance challenges posed by siloed data and prevalent spreadsheet usage
Solution:
- Implemented Snowflake's Data Cloud to create a modern data environment
- Ingested constituent data from Salesforce to support development and alumni relations teams
- Utilized Tableau and Snowflake to streamline the delivery of timely insights to development officers
- Leveraged Snowflake Marketplace to enrich data and build sophisticated models
- Improved data governance and data literacy through unified data in the Data Cloud
Results:
- Enhanced donor and alumni engagement by enriching and analyzing constituent data
- Empowered development officers with embedded Tableau dashboards to identify outreach opportunities, inform donor conversations, and track interactions
- Assessed the impact of advancement efforts and optimized development officers' portfolios using donor engagement data
- Advanced data literacy through proactive data governance and regular data task force meetings
- Improved Salesforce ETL throughput by 4x using AWS AppFlow and Snowpark for Python, reducing load times and vendor dependency
- Enabled exploratory AI work by bringing in external data sources to enrich data and identify insights
Read the full case study.
Take the next step towards simplifying your data platform
Snowpark introduces a suite of powerful capabilities that can greatly enhance modern data platforms. It bridges the gap between complex data engineering, data science tasks, and the scalable, secure environment of Snowflake, thereby offering a robust solution for managing and analyzing big data.
Snowpark simply enables data professionals to focus on deriving insights and value from data, rather than having to manage the underlying infrastructure and data processing complexities. If you’re interested in starting your data journey with us or simply modernizing your existing data platform and migrating to Snowflake, contact us directly through this contact form, and we’ll be in touch promptly to schedule a free consultation.
