decentral1se/iroh-go
1
1
Fork 1
Code Issues 5 Pull Requests 1 Actions Activity

Support io.Writer / io.Reader #14

New Issue
Open
opened 2026-06-29 14:58:11 +00:00 by cblgh · 5 comments
cblgh commented 2026-06-29 14:58:11 +00:00
Collaborator
Copy Link
Copy Source

Go's io.Writer and io.Reader have massive support across the ecosystem. In some far-off iroh-go future, it would be amazing if we could somehow wrangle in these interfaces and connect them to the ffi plumbing. This seems hard though? It is also not blocking anything right now. But working with these interfaces would make a lot of small papercuts vanish.

The problems are two-fold: signature mismatches and also implementation expectation mismatches. Currently there are some slight signature mismatches which means iroh-go won't conform to either interface. More importantly, the expected behaviour of calling e.g. RecvStream.Read as an io.Reader very likely does not match the ecosystem expectation of calling io.Reader: Read(p []byte) (n int, err error).

Anyway, far future stuff! Maybe we never get there, even then things here are still very useful :) One type of solution would be to write a middle layer that communicates with the ffi on one hand, and on the other hand has data structures and implementations which fulfill all the go io:{Read, Write}er interface expectations. (Maybe what's somewhat happening already? I haven't peeked too much under the hood yet!!)

Go's io.Writer and io.Reader have massive support across the ecosystem. In some far-off iroh-go future, it would be amazing if we could somehow wrangle in these interfaces and connect them to the ffi plumbing. This seems hard though? It is also not blocking anything right now. But working with these interfaces would make a lot of small papercuts vanish. The problems are two-fold: signature mismatches and also implementation expectation mismatches. Currently there are some slight signature mismatches which means iroh-go won't conform to either interface. More importantly, the expected behaviour of calling e.g. [`RecvStream.Read`](https://godocs.io/git.coopcloud.tech/decentral1se/iroh-go#RecvStream.Read) as an `io.Reader` very likely does not match the ecosystem expectation of calling [`io.Reader: Read(p []byte) (n int, err error)`](https://godocs.io/io#Reader). Anyway, far future stuff! Maybe we never get there, even then things here are still very useful :) One type of solution would be to write a middle layer that communicates with the ffi on one hand, and on the other hand has data structures and implementations which fulfill all the go `io:{Read, Write}er` interface expectations. (Maybe what's somewhat happening already? I haven't peeked too much under the hood yet!!)
❤️ 1
decentral1se commented 2026-06-29 17:46:54 +00:00
Owner
Copy Link
Copy Source

It's great that you're running into this already! I don't think it'd be too hard actually, logistically speaking. Most repos that make use of FFI / uniffi end up writing a wrapper in between bindings and consumers, so this is pretty standard.

I'm playing around with a wrapper that makes error handling less awful in #10 which should be a first example of how it works with wrapping bindings for re-export.

Could you (whenever) sketch out your ideal API based on the code you're writing and what you'd like to encapsulate into the Reader/Writer? It would be a nice milestone to work towards 🧗‍♀️

It's great that you're running into this already! I don't think it'd be too hard actually, logistically speaking. Most repos that make use of FFI / uniffi end up writing a wrapper in between bindings and consumers, so this is pretty standard. I'm playing around with a wrapper that makes error handling less awful in https://git.coopcloud.tech/decentral1se/iroh-go/issues/10 which should be a first example of how it works with wrapping bindings for re-export. Could you (whenever) sketch out your ideal API based on the code you're writing and what you'd like to encapsulate into the `Reader`/`Writer`? It would be a nice milestone to work towards 🧗‍♀️
❤️ 1
cblgh commented 2026-06-29 19:41:50 +00:00
Author
Collaborator
Copy Link
Copy Source

Could you (whenever) sketch out your ideal API based on the code you're writing and what you'd like to encapsulate into the Reader/Writer? It would be a nice milestone to work towards 🧗‍♀️

That's a super idea! So concretely where I had the idea was in working with a smol wire protocol and needing to send prefixed length byte streams. I was looking to use encoding/binary:Varint but then noticed the Read and Write methods. If I could've passed in a reader / writer from iroh-go to these, I would've (I think?) been saved a little confusion and wrangling!

I'll have to return to this for an a good API sketch because I'm still finding my way. But an early minimal idea maybe:

io.Read(p []byte) (n int, err error)

// usual connection stuff, no change
  accepting, err := incoming.Accept()
  check(err)
  conn, err := accepting.Connect()
  check(err)
  stream, err := conn.AcceptBi()
  check(err)

  recv := stream.Recv()
  // here's the difference:
  data := make([]byte, 1024)
  for {
     bytesRead, err := recv.Read(data)
     check(err)
     fmt.Println(data[:bytesRead])
     // alternatively, since RecvStream implements io.Read, i could also do:
     binary.Read(recv, binary.LittleEndian, data)
   }

io.Writer.Write(p []byte) (n int, err error)

// usual stuff, no change
  accepting, err := incoming.Accept()
  check(err)   
  conn, err := accepting.Connect()
  check(err)
  stream, err := conn.AcceptBi()
  check(err)
  
  send := stream.Send()
  // here's the difference
  buf := []byte("my cool and very long message")
  // send data
  bytesWritten, err := send.Write(data)
  check(err)
  fmt.Printf("wrote %d bytes\n")
  
 // alternatively, since SendStream implements io.Write, i could also do:
binary.Write(send, binary.LittleEndian, data)

But maybe it's hard / not worth the wrangling to change Write and Read on these core iroh abstractions, so perhaps a compromise might be to wrap what already exists à la NewReader, NewWriter conventions:

  irohgo.NewWriter(stream *iroh.BiStream) -> returns some struct that implements io.Writer
  irohgo.NewReader(stream *iroh.BiStream) -> returns some struct that implements io.Reader

  // which would give us 
  recv := irohgo.NewReader(stream) // stream is *iroh.BiStream; wraps *RecvStream
  data := make([]byte, 1024)
  bytesRead, err := recv.Read(data)
   
  // and 
  send := irohgo.NewReader(stream) // stream is *iroh.BiStream; wraps *SendStream
  buf := []byte("my cool and very long message")
  bytesWritten, err := send.Write(data)
> Could you (whenever) sketch out your ideal API based on the code you're writing and what you'd like to encapsulate into the Reader/Writer? It would be a nice milestone to work towards 🧗‍♀️ That's a super idea! So concretely where I had the idea was in working with a smol wire protocol and needing to send prefixed length byte streams. I was looking to use [`encoding/binary:Varint`](https://pkg.go.dev/encoding/binary#Varint) but then noticed the [`Read`](https://godocs.io/encoding/binary#Read) and [`Write`](https://godocs.io/encoding/binary#Write) methods. If I could've passed in a reader / writer from `iroh-go` to these, I would've (I think?) been saved a little confusion and wrangling! I'll have to return to this for an a good API sketch because I'm still finding my way. But an early minimal idea maybe: `io.Read(p []byte) (n int, err error)` ```golang // usual connection stuff, no change accepting, err := incoming.Accept() check(err) conn, err := accepting.Connect() check(err) stream, err := conn.AcceptBi() check(err) recv := stream.Recv() // here's the difference: data := make([]byte, 1024) for { bytesRead, err := recv.Read(data) check(err) fmt.Println(data[:bytesRead]) // alternatively, since RecvStream implements io.Read, i could also do: binary.Read(recv, binary.LittleEndian, data) } ``` `io.Writer.Write(p []byte) (n int, err error)` ```golang // usual stuff, no change accepting, err := incoming.Accept() check(err) conn, err := accepting.Connect() check(err) stream, err := conn.AcceptBi() check(err) send := stream.Send() // here's the difference buf := []byte("my cool and very long message") // send data bytesWritten, err := send.Write(data) check(err) fmt.Printf("wrote %d bytes\n") // alternatively, since SendStream implements io.Write, i could also do: binary.Write(send, binary.LittleEndian, data) ``` But maybe it's hard / not worth the wrangling to change `Write` and `Read` on these core iroh abstractions, so perhaps a compromise might be to wrap what already exists à la `NewReader`, `NewWriter` conventions: ```golang irohgo.NewWriter(stream *iroh.BiStream) -> returns some struct that implements io.Writer irohgo.NewReader(stream *iroh.BiStream) -> returns some struct that implements io.Reader // which would give us recv := irohgo.NewReader(stream) // stream is *iroh.BiStream; wraps *RecvStream data := make([]byte, 1024) bytesRead, err := recv.Read(data) // and send := irohgo.NewReader(stream) // stream is *iroh.BiStream; wraps *SendStream buf := []byte("my cool and very long message") bytesWritten, err := send.Write(data) ```
🚀 1
decentral1se commented 2026-06-30 09:45:11 +00:00
Owner
Copy Link
Copy Source

Thanks @cblgh 👏 And last thing, can you copy/pasta in the "current code"? You wrote

// here's the difference

But I miss the code of what you're doing right now? Then we have the full picture.

Thanks @cblgh 👏 And last thing, can you copy/pasta in the "current code"? You wrote > // here's the difference But I miss the code of what you're doing right now? Then we have the full picture.
cblgh commented 2026-06-30 10:14:38 +00:00
Author
Collaborator
Copy Link
Copy Source

Sure! I just refactored the relevant bits into a util so that I could reuse the same logic on client & server

MAY CONTAIN BUGS :) :)

func WriteData(data []byte, send *iroh.SendStream) {
	// NOTE (2026-06-30): bit of thrashing around having to instantiate sizebuf all the time
	sizebuf := make([]byte, binary.MaxVarintLen64)
	// calculate varint length of data
	// prefix data with the length
	fmt.Println("wrote", string(data))
	wrote := binary.PutVarint(sizebuf, int64(binary.Size(data)))
	sizebuf = sizebuf[:wrote]
	// prepend varint length to data
	data = append(sizebuf, data...)
	// fmt.Println("data", data, "size", size, "len(data)", len(data))
	// send data
	err := send.WriteAll(data)
	Check(err)
}

const frameSize = 1024 // NOTE (2026-06-30): arbitrarily chosen value, can be tweaked if things are not working
func ReadBytes(recv *iroh.RecvStream, ch chan<- []byte, done chan struct{}) {
	for {
		frame, err := recv.Read(frameSize)
		if checkForTimeout(err) {
			break
		}
		ch<-frame
	}
	close(done)
}

const SENTINEL = -1024
func ProcessRead (recv *iroh.RecvStream, processed chan<- []byte, done chan struct{}) {
	var size int64 
	size = SENTINEL
	dataCh := make(chan []byte)
	data := make([]byte, 0)
	go ReadBytes(recv, dataCh, done)
	for {
		select {
		case <-done:
			return
		case frame := <- dataCh:
			data = append(data, frame...)
			fmt.Println("recv", string(frame), cap(frame))
		default:
			// dont block
		}
		// if we don't have any data we instead of continuing the loop block on receiving from dataCh
		if len(data) == 0 {
			frame := <-dataCh
			data = append(data, frame...)
		}
		bytesread := 0
		if (size == SENTINEL) {
			size, bytesread = binary.Varint(data)
			if bytesread > 0 {
				data = data[bytesread:]
			} else {
				panic("handlReading: varint wasn't where we expected it")
			}
		}
		if len(data) >= int(size) {
			processed <-data[:size]
			data = data[size:]
			size = SENTINEL
		}
	}
}
func checkForTimeout(err error) bool {
	// handle timeout
	if err != nil {
		irohErr := err.(*iroh.IrohError)
		errMsg := irohErr.Message()
		if strings.HasPrefix(errMsg, "ConnectionLost") {
			fmt.Println("lost connection")
			return true
		} else {
			// panics and exits bc unhandled err
			Check(err)
		}
	}
	return false
}

func Check(err error) {
	if err != nil {
		irohErr := err.(*iroh.IrohError)
		panic(irohErr.Message())
	}
}

hope it helps!

Sure! I just refactored the relevant bits into a util so that I could reuse the same logic on client & server MAY CONTAIN BUGS :) :) ```golang func WriteData(data []byte, send *iroh.SendStream) { // NOTE (2026-06-30): bit of thrashing around having to instantiate sizebuf all the time sizebuf := make([]byte, binary.MaxVarintLen64) // calculate varint length of data // prefix data with the length fmt.Println("wrote", string(data)) wrote := binary.PutVarint(sizebuf, int64(binary.Size(data))) sizebuf = sizebuf[:wrote] // prepend varint length to data data = append(sizebuf, data...) // fmt.Println("data", data, "size", size, "len(data)", len(data)) // send data err := send.WriteAll(data) Check(err) } const frameSize = 1024 // NOTE (2026-06-30): arbitrarily chosen value, can be tweaked if things are not working func ReadBytes(recv *iroh.RecvStream, ch chan<- []byte, done chan struct{}) { for { frame, err := recv.Read(frameSize) if checkForTimeout(err) { break } ch<-frame } close(done) } const SENTINEL = -1024 func ProcessRead (recv *iroh.RecvStream, processed chan<- []byte, done chan struct{}) { var size int64 size = SENTINEL dataCh := make(chan []byte) data := make([]byte, 0) go ReadBytes(recv, dataCh, done) for { select { case <-done: return case frame := <- dataCh: data = append(data, frame...) fmt.Println("recv", string(frame), cap(frame)) default: // dont block } // if we don't have any data we instead of continuing the loop block on receiving from dataCh if len(data) == 0 { frame := <-dataCh data = append(data, frame...) } bytesread := 0 if (size == SENTINEL) { size, bytesread = binary.Varint(data) if bytesread > 0 { data = data[bytesread:] } else { panic("handlReading: varint wasn't where we expected it") } } if len(data) >= int(size) { processed <-data[:size] data = data[size:] size = SENTINEL } } } func checkForTimeout(err error) bool { // handle timeout if err != nil { irohErr := err.(*iroh.IrohError) errMsg := irohErr.Message() if strings.HasPrefix(errMsg, "ConnectionLost") { fmt.Println("lost connection") return true } else { // panics and exits bc unhandled err Check(err) } } return false } func Check(err error) { if err != nil { irohErr := err.(*iroh.IrohError) panic(irohErr.Message()) } } ``` hope it helps!
🚀 1
decentral1se commented 2026-07-02 06:47:31 +00:00
Owner
Copy Link
Copy Source

Reference:

https://git.coopcloud.tech/cblgh/iroh-tracker/src/branch/main/util/util.go

Reference: > https://git.coopcloud.tech/cblgh/iroh-tracker/src/branch/main/util/util.go
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
cblgh decentral1se (d1)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: decentral1se/iroh-go#14
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?