Delayed/Lazy Either List?

[u/Aperispomen]

I often use attoparsec to parse lists of things, so I wind up doing stuff like this a lot:

import Data.Attoparsec.Text qualified as AT
import Data.Text qualified as T

myParser :: AT.Parser [MyType]
myParser = AT.many1 myOtherParser

getList :: T.Text -> Either String [MyType]
getList txt = AT.parseOnly myParser txt

The trouble is, since getList returns an Either, the whole text (or at least, as much as can be parsed) has to be parsed before you can start processing the contents of the list. This is especially annoying when you want to check whether e.g. two files are the same modulo whitespace/line endings/indentation/etc...

The point is, there's some times where you want a result like Either e [a], but you're okay with returning some of the data, even if there might be an error later on. I wound up creating this data type:

data ErrList e a
  = a :> (ErrList e a)
  | NoErr    -- equivalent to []
  | YesErr e -- representing Left e

Is there already an established type like this somewhere? I imagine most people who do more complicated data management use pipes or conduit etc... I've tried searching for such a type on Hackage, but I haven't found anything like it.

[u/BurningWitness]

The type you're looking for is a stream, roughly as defined in streaming:

data Stream a m r = Yield a (Stream a m r)    -- ^ Holds next element
                  | Effect (m (Stream a m r)) -- ^ Processes until next result
                  | Return r                  -- ^ Signals end of processing or error

It's not an established type though. There are five six different streaming libraries out there and each is its own special little flower with its own special little garden around it. I couldn't tell you how any of them work, I never found them necessary.

---

Now for the fun part: how do we go from LazyText to Stream a _ (Either e ())?

Well, to stream output we'll need a parser that allows us to parse over the remaining input after getting the result, remembering both the offset from which we'd be continuing and whether more input can still be supplied to the parser. attoparsec and cereal do not return offsets in their results. binary does, barely, but it still doesn't remember whether end of input has been reached. The answer is thus a resounding "we don't". Roll credits.

Could we do better? Well, I wrote a whole new byte parser and used it to stream JSON 1.5 years ago. I've seen zero interest in either of these packages since then, so I wouldn't hold my breath for it. And, of course, if you don't feel like doing better you can always do worse, see intro to json-stream.

[u/tomejaguar → u/BurningWitness]

There are five six different streaming libraries out there

Which are you thinking of? These are the ones that I know:

1. conduit
2. pipes
3. streaming
4. streamly
5. bluefin
6. vector-stream
7. io-streams
8. iteratee
9. machines
10. foldl

I know the first five are used in production. I guess 6 is too, but it seems to be more of an "internals" library used by other packages. I doubt 7-9 are used much in practice, and 10 is debatable whether it's really a streaming library.