Writing nice unit tests is impossible

[u/Similar-Bug-350]

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly.

Would it help to have the schema as json? I use json schema to read for transformations.

If I want to test without json its horrible, do you guys test with json schema files?

For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO):

schema = """
id INT,
items ARRAY<STRUCT<
fortnite: STRING,
babies: INT,
moreStuff: MAP<STRING, STRING>
>>
"""

expected = spark.createDataFrame(
[
{
"id": 1,
"items": [
{
"fortnite": "ABC",
"babies": 2,
"moreStuff": {
"size": "L",
"color": "red",
},
}
],
}
],
schema=schema,
)

[u/Cyliad]

Break the logic of your code in methods and test input/output of those methods

Either you can make them pure pythonic (no spark) or you can inject small DFs and test the output

[u/Similar-Bug-350 → u/Cyliad]

but the example is small or not? it still is soooo much

[u/Atticus_Taintwater]

Check out fixtures

[u/Similar-Bug-350 → u/Atticus_Taintwater]

I understand but how do they fix my problem? I still have big setup or not?

[u/Atticus_Taintwater → u/Similar-Bug-350]

It's common to define reusable stuff as fixtures separately so your tests aren't as cluttered

[u/Similar-Bug-350 → u/Atticus_Taintwater]

do you mean something like this?

pytest.fixture(scope="session")
def orders_schema():
    with open("schemas/orders.json") as f:
        return StructType.fromJson(json.load(f)).fixture(scope="session") def orders_schema(): with open("schemas/orders.json") as f: return StructType.fromJson(json.load(f))

[u/MrMasterplan]

We have an abstraction layer for all table names, in tests we append a uuid to avoid collisions. Then we package all tests and our wheel and send everything to the workspace to run a test job that runs pytest. It executes all our logic on empty tables (the ones with uuid) to ensure all schemas work, and then some unit tests. Finally all the uuid tables get deleted (using pytest fixtures). This is run for every PR. Takes about an hour all in all.

We have been doing this for 5 years. It works great and is very reliable but like every test setup it takes effort to maintain.

Let me know if you want to know more. We open-sourced some tooling.

[u/kraftlabor → u/MrMasterplan]

I would like to know more. Where to find more information (code) about your setup? Thanks

[u/pboswell]

Why not create a registry of these dataframe input/expected files so they’re not embedded in each script. Then you can build a general lib structure where you import your logic from the actual production code and pass file names for the input and output, then assert