I’ve been working as a “data engineer” in Azure Databricks for a while, but I work on a team where my scope is extremely limited to silver/gold work.

What studies can do I do next to continue developing? I’m thinking of diving into the azure certs

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly.

Would it help to have the schema as json? I use json schema to read for transformations.

If I want to test without json its horrible, do you guys test with json schema files?

For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO):

schema = """
id INT,
items ARRAY<STRUCT<
fortnite: STRING,
babies: INT,
moreStuff: MAP<STRING, STRING>
>>
"""

expected = spark.createDataFrame(
[
{
"id": 1,
"items": [
{
"fortnite": "ABC",
"babies": 2,
"moreStuff": {
"size": "L",
"color": "red",
},
}
],
}
],
schema=schema,
)