---
esco_uri: http://data.europa.eu/esco/skill/4463a721-69f3-413d-8321-43e3af13a4f1
esco_label: "use databases"
relation: essential
onet_soc: 15-2051.00
source: stackexchange
confidence: high
qa_count: 5
generator: gemma3:27b (prompt-designed and spot-checked by Claude)
generated: 2026-07-10
---

# use databases — Data Scientist

For a Data Scientist, 'using databases' isn’t just about running SELECT statements; it's core to the entire workflow. Daily tasks involve extracting data for analysis (often complex joins and aggregations), transforming that data *within* the database using SQL or stored procedures before bringing it into Python/R, and then potentially loading processed results back in for reporting or model serving. Think building data pipelines – databases are frequently the central hub. Common tools include SQL variants (PostgreSQL, MySQL, Snowflake, BigQuery), NoSQL options like MongoDB for flexible schemas, and cloud-based database services.

## Praxiswissen (Stack Exchange, CC-BY-SA 4.0)

**The user questions the need for databases when they can easily store and retrieve data using simple file serialization (like JSON), finding it faster and more straightforward.**

While basic data storage works initially, databases offer significant advantages as projects grow in complexity. They provide powerful tools for querying, relating, and analyzing data efficiently, ensuring consistency and reliability through features like ACID compliance and fault tolerance. Utilizing a database leverages decades of development and optimization, offering scalability and concurrent access that simple file-based solutions cannot easily match.

*Source: [Why use a database instead of just saving your data to disk?](https://softwareengineering.stackexchange.com/q/190482) — answer by Robert Harvey, CC-BY-SA 4.0*

**The user is deciding whether to store user-uploaded files (1-10MB) directly within their MySQL database or store them on the filesystem and reference them with paths in the database, concerned about performance impacts.**

While storing files *in* a database offers transactional consistency and simplifies backups, it significantly increases database size and maintenance complexity. Larger databases require more resources, specialized knowledge to manage, and can hinder portability. Storing files on the filesystem avoids these issues, offering simpler access and reducing database overhead, but requires careful management of file synchronization and potential orphan records.

*Source: [Is it a bad practice to store large files (10 MB) in a database?](https://softwareengineering.stackexchange.com/q/150669) — answer by Thomas, CC-BY-SA 4.0*

**The question asks about the differences between `localStorage` and `indexedDB` for client-side data storage, and when to use each.**

While both technologies offer offline data persistence, they are designed with different scales in mind. `localStorage` is simple, synchronous, and best suited for small amounts of string-based data like user preferences or session tokens. `indexedDB` is more complex but handles larger datasets efficiently through an asynchronous API, indexing capabilities, and support for various data types; it's a better choice when you need to store and query substantial structured information.

*Source: [How is localStorage different from indexedDB?](https://softwareengineering.stackexchange.com/q/219953) — answer by yannis, CC-BY-SA 4.0*

**The user routinely adds an auto-incrementing integer primary key named 'id' to every database table for unique row identification, and is questioning if this practice has drawbacks or when it might be unnecessary.**

Having a guaranteed unique identifier per row is generally beneficial. While adding such a key does incur minor overhead in storage and index maintenance, the advantages of simplified data access and relationship management usually outweigh these costs. Consider whether a natural key already sufficiently guarantees uniqueness before automatically adding an 'id' field.

*Source: [Is it good practice to always have an autoincrement integer primary key?](https://softwareengineering.stackexchange.com/q/328458) — answer by GrandmasterB, CC-BY-SA 4.0*

**The user is designing a database for a task assignment system and struggling with how to represent many-to-many relationships between people and tasks without creating unwieldy table structures with numerous redundant columns.**

Instead of storing lists of IDs within tables or adding excessive columns, the best practice is to create a separate 'junction' or 'linking' table. This new table establishes explicit relationships by referencing primary keys from both related tables, adhering to database normalization principles. This approach allows for flexible and scalable many-to-many associations without data duplication or management overhead.

*Source: [Is it ever okay to use lists in a relational database?](https://softwareengineering.stackexchange.com/q/381460) — answer by whatsisname, CC-BY-SA 4.0*

## Weitere Anreicherung

Stage-2 source for future practitioner grounding: arXiv cs.SE (software-engineering preprints).
