In my conversations with data leaders across industries, one thing is clear: data engineering is under pressure like never before.
We’re seeing explosive growth in data variety, real-time demands, and the need to operationalize AI. But the patterns and architectures many teams are currently dependent on, are just not able to keep up.
As someone who’s worked closely with hundreds of data teams, I see engineers facing five big challenges today. Let’s look at these challenges and the solutions I believe will shape the future of the field..
1. Ingestion at Scale
Scaling ingestion isn’t just about throughput; it’s about reliability, flexibility, and cost control.
Too often, I see teams fighting:
- Schema drift in streaming data that breaks fragile pipelines
- Real-time ingestion of structured and unstructured data creating operational chaos
- Over-provisioned cloud resources driving runaway costs
Where I see teams succeeding:
- Adopting schema-on-read architectures like Delta Lake or Apache Iceberg to tolerate schema drift without constant rewrites.
- Using progressive batching to ingest raw data in micro-batches, transforming only critical metrics in real time, balancing cost and speed.
- Leveraging platforms (like Knowi) that virtualize ingestion across 40+ sources without pre-loading, reducing pipeline failures and simplifying operational overhead.
2. The Warehouse vs. Lake Debate
For years, data warehouses and lakes served different masters. Warehouses excel at structured, regulated reporting. Lakes are built for exploratory, unstructured, and real-time data.
But maintaining both introduces:
- Complexity
- Redundant pipelines
- Governance headaches
What I see coming:
These distinctions will blur. Warehouses will evolve into performance-optimized sandboxes for reporting, while lakes serve as discovery layers for real-time and unstructured data. A true lakehouse approach will enforce governance contracts between them.
Key innovation:
Metadata will drive automatic data placement, hot data in warehouses, cold in lakes. Tools that query both directly without migration eliminate redundant pipelines while maintaining governance.
3. Orchestration Complexity
Data orchestration used to be about simple DAGs on a schedule. Those days are over.
Modern pipelines must handle:
- Complex dependencies
- Real-time adaptations
- Data quality at scale
Too often, I see teams stuck with manual orchestration that leads to brittle systems and endless maintenance.
What advanced teams are adopting:
- Event-driven workflows that respond instantly to changes.
- AI-powered, self-healing pipelines that detect and resolve failures.
- Multi-domain state management to ensure consistent governance.
Alternative approach:
Instead of orchestrating complex ETL jobs, platforms like Knowi let you query data in place across diverse sources. That means no heavy dependency chains, lower latency, and built-in data quality checks.
4. Choosing the Right Processing Engine
I’ve seen many teams agonize over choosing between Spark, Flink, or other engines. It’s not just about speed, it’s about matching the engine to the use case.
- Spark excels at large-scale batch processing.
- Flink’s event-driven model is perfect for real-time, low-latency analytics.
The integration headache:
Many platforms force you to move data into their environment for processing, introducing complexity and delay.
What we advocate:
Querying in place. Platforms like Knowi support querying directly in databases, lakes, or APIs without ETL overhead. This reduces latency and simplifies architecture—enabling truly real-time analytics without pipeline runs.
5. Operationalizing AI in the Stack
Too often, I see AI tacked onto data stacks as:
- Isolated models
- Third-party APIs
- Opaque black boxes
This creates security risks, integration friction, and governance nightmares.
What I see as the path forward:
- On-premise AI foundations that keep all processing inside the organization for GDPR/CCPA compliance.
- Integrated generative AI to accelerate everything from query generation to dataset creation.
- Natural Language Interfaces that let users ask complex questions in plain English.
- Automated insights for anomaly detection and next-best-action recommendations.
Our approach at Knowi:
We embed a secure, on-premise small language model that supports conversational analytics, automated plain-English summaries, and unified querying across structured data and unstructured documents. Imagine asking, “How do contract terms impact sales?” and getting an answer with transparent, auditable sources.
Conclusion
Data engineering isn’t getting simpler. But it is getting more advanced. The best strategies don’t fight complexity with more complexity, they abstract it away
By adopting:
- Schema-on-read ingestion
- Unified lakehouse governance
- In-place querying
- Advanced orchestration
- Deeply integrated AI
you can build a data stack that is resilient, flexible, and cost-effective.
