Rework & separate renderers

[ACrazyTown]

Here we go, first step towards a renderer overhaul. Looks like a pretty big and scary change but 99% of this is just moving stuff around.

Shout out goes to Beeblerox & Austin East, a lot of this is based on the original renderer overhaul branch, I'm just porting bits and pieces to modern Flixel.

Changes

FlxRenderer

Adds a base FlxRenderer class, accessible via the global FlxG.renderer, which serves as the base for all rendering functionality. Renderer implementations extend it and implement the required methods.

The blitting and draw quads renderers have been ported to FlxBlitRenderer and FlxQuadRenderer, respectively.

Since the renderer is a global thing, and not per-camera anymore, it works slightly differently. Before any calls to drawing commands you need to call FlxG.renderer.begin(camera);. This will be done internally by Flixel during a sprite's draw phase, but it's something to keep in mind if you're doing something out of the ordinary!

FlxCameraView

Adds a base FlxCameraView class. Like with renderers, different implementations extend the base class and add onto it. Camera views mainly just hold per-camera rendering related objects, stuff like OpenFL sprites and whatnot.

To avoid breaking changes, you can get a typed reference to the camera view using the camera.viewBlit and camera.viewQuad shortcuts. Use this to reference stuff like camera.flashSprite or camera.canvas.

Other previously established stuff

Most rendering methods from FlxCamera have been deprecated. If you need to issue drawing commands manually, use the FlxG.renderer API instead.

I say most because some batch related methods (e.g. startQuadBatch()) have been left as-is. I plan to tackle these at a later point and in a different PR.

---

I'm opening this as a DRAFT, because:

• This needs to be tested thoroughly to make sure I didn't accidentally break something
• Would be nice to add docs to a bunch of stuff
• I don't know what to do with a bunch of private internal variables in FlxCamera and alike. What's Flixel's way of handling this? Should I simply get rid of them, or deprecate them and make them point to where they were moved?

TODO:

• flixel/addons/display/FlxShaderMaskCamera.hx:222 -- Field fill should be declared with overload since it was already declared as overload in superclass
• Documentation
• Figure out what to do with deprecated private functions/vars
• Better function names?

[ACrazyTown]

I decided to deprecate all the internals and make them point to where they were moved, to avoid breaking anything.

Addons and UI will need to be updated to avoid warnings, I'll get to that in a bit. Addons specifically seems to have an issue because FlxShaderMaskCamera overrides camera.fill(), it'll need to be updated to also add overload extern, hopefully that's not a blocker

I think this is in a good enough of a state now for a review, so I'll undraft

[ACrazyTown]

Some more notes and thoughts. I'm currently playing around trying to implement an OpenGL renderer to really test the flexibility of this system.

Overloading methods in FlxCameraView won't work:

For stuff like FlxMaterial and FlxTrianglesData, the function signatures of the core rendering functions need to be changed. To avoid breaking changes, I was just going to do this with overloads, but that didn't work out because overloads need to be inlined and you can't override inline functions.

I got around this by deprecating drawPixels() and copyPixels() for draw() and copy() in FlxCameraView, but I couldn't do the same in FlxCamera because it inherits FlxBasic.draw(). I ended up overloading the camera's drawPixels() and copyPixels() to call the new methods in camera view. Pretty gross solution IMO but I can't think of a better way without a breaking change.

Unifying renderBlit with other renderers

I did some work related to this in 3cd97d9, but there's still a couple places where we have to do things differently depending on the renderer, notably drawPixels() and copyPixels():

flixel/flixel/FlxCamera.hx

Lines 743 to 744 in c32ab91

	public function drawPixels(?frame:FlxFrame, ?pixels:BitmapData, matrix:FlxMatrix, ?transform:ColorTransform, ?blend:BlendMode, ?smoothing:Bool = false,
	?shader:FlxShader):Void

When the blitting renderer is used, pixels will be used for the rendering, otherwise frame is used. I wonder if it's somehow possible to avoid this special behavior and just pass in one thing, without having to know what renderer is used. The user should just have to call camera.drawPixels(...); once, and the underlying implementation should take care of any special quirks.

Isolating direct access to renderer

Flixel shouldn't access any renderer implementations directly from common code, it should be done through the renderer abstraction. First thing that comes to mind is that we need to abstract way maxTextureSize

EDIT: This could be done via the RenderFrontEnd via some method like FlxG.render.getMaxTextureSize()

EDIT 2: This is no longer an issue with recent changes, see next comment

[ACrazyTown]

Originally this was just a port of the renderer abstraction by Beeblerox that I ported over to modern Flixel. As I was playing around with it, I found that there were certain things I wasn't too fond of, specifically the fact that access to the renderer was only possible through cameras, so here goes an attempt at a V2.

FlxCameraView has been demoted, and no longer handles talking to the renderer. This is now done by the global FlxRenderer (Accessible via FlxG.renderer) and its implementations. FlxCameraView is still around, though it now mainly just stores the various objects needed per-camera for rendering (e.g. the flash sprites and whatever).

Considering that the renderer is now global, I've also integrated some of the helpers mentioned in #3527 directly into FlxRenderer.

I also wonder if it'd be worth deprecating the drawing methods in FlxCamera, and pointing users who need advanced control over the renderer to use FlxG.renderer instead.

[ACrazyTown]

I think this is now in a reviewable state. No clue why CI is failing tho

[Geokureli]

Apparently @:bypassAccessor doesn't work with setters> I could have sworn it did. Gonna rethink this a bit

[ACrazyTown]

Nice work, thanks for helping out with this. I like most of the changes here, though I do have some concerns/questions.

Re: No.3

I think moving render() and clear() make sense as well, though I have some mixed feelings about moving fill() and the debug drawing methods as they're rendering commands and they interact with the renderer (eg. fill() just draws a colored quad that covers the screen). This will make more sense with the OpenGL renderer where objects are tied to the renderer rather than to cameras. See for example how I was handling fill() while messing around with OpenGL: https://github.com/ACrazyTown/flixel/blob/feat/opengl-renderer-2/flixel/system/render/gl/FlxGLRenderer.hx#L112-L121

"This allows internal methods used by these to be private, rather than @:noCompletion public"

I don't think I completely understand what you mean. If you mean that view.fill() will call some private renderer.fill() method then that makes sense, I agree, and ignore this entire paragraph lol.

Re: Decouple FlxCameraView from FlxCamera

I've been thinking about this too. I think it's doable to have some arbitrary buffer draw commands can be executed on (though perhaps this should just be FlxRenderTexture?). For drawing sprites and whatnot I think we'd still need a camera though, because of how the draw phase works.

Food for thought, Beeblerox's FlxRenderTarget

Re: Obfuscate openfl.display.Graphics

Hmm, some food for thought, the renderer overhaul had the FlxDraw class as a way to draw hardware accelerated shapes. It's not a direct equivalent to Graphics as it lacks features but performance wise it should be much better. Perhaps we could unify the two in some way, use FlxDraw when drawing to the screen and Graphics when stamping to a texture, or something.

Re: "Quad" vs "Tile"

This bothered me as well, but I couldn't decide what to do. DRAW_QUADS would be a more accurate name as that's the method used (graphics.drawQuads()). DRAW_TILES is the older method which was removed in like OpenFL 4.0 (and it's why Flixel wasn't compatible with later OpenFL versions for a few years). I'm not sure why DRAW_TILES wasn't renamed before, perhaps it was an oversight?

Personal concerns

Breaking core function signatures

This is not completely related to these specific changes, but I thought I'd mention it so that we could future proof stuff before things are set in stone. Ideally I was hoping that the OpenGL renderer, which I'd implement some time after this is merged, wouldn't break anything. That way it could be locked behind a compiler flag in Flixel 6.x while it gets fully polished and tested, and made default in Flixel 7. (and perhaps we drop the DRAW_QUADS renderer then?)

One issue with this is that the renderer overhaul comes with some function signature changes. As an example, here's what drawTriangles() could look like with the changes:

// Old
function drawTriangles(graphic:FlxGraphic, vertices:DrawData<Float>, indices:DrawData<Int>, uvtData:DrawData<Float>,
		?colors:DrawData<Int>, ?position:FlxPoint, ?blend:BlendMode, repeat:Bool = false, smoothing:Bool = false, ?transform:ColorTransform,
		?shader:FlxShader):Void {}

// New
function drawTriangles(graphic:FlxGraphic, data:FlxTrianglesData, material:FlxMaterial, matrix:FlxMatrix, ?transform:ColorTransform):Void {}

It's a pretty significant, and IMO welcome change. I've been thinking over how to implement it in a non-breaking way though. Everything I've thought of thus far has some flaw:

• Deprecate old, and introduce new methods
  • This would be the easiest way, but what do we call them? drawPixels() and copyPixels() are in my opinion kinda strange already, and I'm not really good with naming stuff lol. Maybe something like drawQuad()? For triangles perhaps we could consider something like drawMesh() but this would create an inconsistency where everything else refers to triangles but the renderer method refers to it as a mesh.
• Overloads
  • I'm not sure if this can even be pulled off due to inheritance. I played around with making the renderer an abstract class a while back in hopes of avoiding this, and while it technically made it possible to overload methods, they'd only be accessible if we had a typed reference to the extending class and not in the base class
• Rework & separate renderers #3539 (comment)
  • A bit ugly, and not really non-breaking, just gives us an excuse to do so.

It's not really crucial that the OpenGL renderer gets into Flixel 6, but I think it'd be really nice, and also would make the transitional period a lot smoother.

[Geokureli]

Breaking core function signatures

One issue with this is that the renderer overhaul comes with some function signature changes.

The secret is it's always gonna be ugly. I tend to make new methods, in your example I'd add drawMaterialTriangles or something.

It's not really crucial that the OpenGL renderer gets into Flixel 6, but I think it'd be really nice, and also would make the transitional period a lot smoother.

The sooner we try to utilize the new system the sooner we'll realize we're doing something wrong. The more we can plan now, the better.

Re: No.3

I think moving render() and clear() make sense as well, though I have some mixed feelings about moving fill() and the debug drawing methods as they're rendering commands and they interact with the renderer (eg. fill() just draws a colored quad that covers the screen). This will make more sense with the OpenGL renderer where objects are tied to the renderer rather than to cameras. See for example how I was handling fill() while messing around with OpenGL: https://github.com/ACrazyTown/flixel/blob/feat/opengl-renderer-2/flixel/system/render/gl/FlxGLRenderer.hx#L112-L121

This is where I'm really struggling to understand the idea. This isn't really how I imagine other global utils are used, and it seems weird, here. Is the idea behind FlxG.renderer that the "screen" is a global object that things draw to, where the views simply determine how world coordinate translate to screen coordinates? I see it very differently, where the screen has a bunch of render targets, with specific layering, orientation, filters and properties. To me objects draw to a render target, regardless of the backend, even in the case of a GL backend, the Flixel dev is choosing which sprites draw to which render target. Perhaps the backend combines that all into one actual screen, but the notion of these being individual things is still important. Thats why filling a view instance via a global call feels weird.

In contrast, say I wanted to fill the screen at an arbitrary time during the draw call, say this had nothing do with a specific view, I simply want the entire screen filled, then I could see using FlxG.renderer.fill(myColor, myMergeAlpha), but that's not what we have here, all of the global rendering functions are specific to a view, and it uses the view's bounds, therefore it makes perfect sense to call view.fill.

"This allows internal methods used by these to be private, rather than @:noCompletion public"

I don't think I completely understand what you mean. If you mean that view.fill() will call some private renderer.fill() method then that makes sense, I agree, and ignore this entire paragraph lol.

Kinda, maybe, idunno...? In the example above fill wouldn't be private, and it's public in your gl example, so that confused me. I also see that your glRenderer's fill uses a shared rect, so the setup kinda makes sense, unlike the blit and quad views which each have a separate displayObject they draw to, gl can just use the same one. To me though, it would make more sense to put a global fillRect(someRect, color, mergeAlpha) in renderer, and have views call that with the camera's margins. Cameras can also have filters(in the future perhaps there could be blending options, too), so if the Gl renderer just draws everything to the screen rather than separating them into specific layers, would we lose that functionality?

I think regardless of what's going on in the backend, object's specified to draw to render targets should call methods on that render target to draw itself. The global util should not need to be an intermediary between those (at least as the default way of drawing to a view). FlxG.renderer should more be for things like project wide settings/configs, and global tools that interact specifically and directly with the backend renderer.

I hope this makes sense, I did the thing where I type all this right before going to bed. I have no concerns or comments on any of your other points

[ACrazyTown]

This is where I'm really struggling to understand the idea. This isn't really how I imagine other global utils are used, and it seems weird, here. Is the idea behind FlxG.renderer that the "screen" is a global object that things draw to, where the views simply determine how world coordinate translate to screen coordinates? I see it very differently, where the screen has a bunch of render targets, with specific layering, orientation, filters and properties. To me objects draw to a render target, regardless of the backend, even in the case of a GL backend, the Flixel dev is choosing which sprites draw to which render target. Perhaps the backend combines that all into one actual screen, but the notion of these being individual things is still important. Thats why filling a view instance via a global call feels weird.

I'll mostly talk about the potential OpenGL renderer here, since it follows the traditional render setup, though the same logic should apply to the other render backends.

The latter interpretation is correct. There can and will be multiple different render targets that all the drawing commands will be executed on. This is how cameras will work, with each camera having its own render texture. The only interaction with the screen (back buffer) will be at the end of the frame, when the camera textures are drawn.

My goal with the global renderer was to keep all the core backend logic centralized, so that we'd keep all the backend OpenGL calls in one place, rather than scattered across multiple classes. There's also benefits to keeping some stuff, like vertex buffers global on a renderer-level rather than per render-target. For example, to batch sprites, we really only need one vertex and index buffer as any state change (such as a render-target swap) will require us to do a draw call and flush the buffer. If we sort our draw commands* to be in order per render target, we can draw everything without a buffer change, which is good because in OpenGL you should keep state changes to a minimum (for reference in the link, a VBO refers to a vertex buffer).

*Stuff like this is why I was considering introducing some sort of draw command buffer, ideally without any additional overhead. I believe modern graphics APIs (i.e. Vulkan, Metal, WebGPU) also use command buffers, so I guess that's future proofing in a way.

We could add convenience methods per view that are basically just something like

// in FlxCameraView
public inline function drawPixels(...)
{
  FlxG.renderer.drawPixels(this, ...);
}

but I still think that all the main rendering stuff should be handled by FlxRenderer implementations.

In contrast, say I wanted to fill the screen at an arbitrary time during the draw call, say this had nothing do with a specific view, I simply want the entire screen filled, then I could see using FlxG.renderer.fill(myColor, myMergeAlpha), but that's not what we have here, all of the global rendering functions are specific to a view, and it uses the view's bounds, therefore it makes perfect sense to call view.fill.

We could allow direct drawing to the screen (back buffer) if the passed in view argument is null. This is usually what happens when you pass in null to some setRenderTarget() method in other higher level graphics libraries. For example, Context3D.setRenderToTexture().

Kinda, maybe, idunno...? In the example above fill wouldn't be private, and it's public in your gl example, so that confused me. I also see that your glRenderer's fill uses a shared rect, so the setup kinda makes sense, unlike the blit and quad views which each have a separate displayObject they draw to, gl can just use the same one. To me though, it would make more sense to put a global fillRect(someRect, color, mergeAlpha) in renderer, and have views call that with the camera's margins. Cameras can also have filters(in the future perhaps there could be blending options, too), so if the Gl renderer just draws everything to the screen rather than separating them into specific layers, would we lose that functionality?

I think regardless of what's going on in the backend, object's specified to draw to render targets should call methods on that render target to draw itself. The global util should not need to be an intermediary between those (at least as the default way of drawing to a view).

I think I cleared up the points here in the paragraph above

FlxG.renderer should more be for things like project wide settings/configs, and global tools that interact specifically and directly with the backend renderer.

Hmm, though FlxG.renderer doesn't expose much in terms of additional functionality. It implements all the higher level drawing methods we've previously had in FlxCamera. I think your point would make sense if we had an actual graphics API built on top. Perhaps something like OpenFL's Context3D, or Ceramic/Clay's GraphicsDriver. This way we could have a single FlxHardwareRenderer or something that just deals with our custom API, and we just swap out the graphics driver. Less duplicating stuff at the cost of less case specific optimizations. It's probably overkill though, and it also wouldn't fare well with the current blit and quad renderer

[Geokureli]

I think my issue is with enforcing the same API across all backends. I'm okay with low-level helpers being in an extension of renderer, but having all these methods defined in the base renderer, and redefined in extensions is going to make things overly rigid, and not well organized in most cases. The interface with the render targets doesn't need to match the interface with the global renderer and if we break that connection I imagine things will go much more smoothly.

My goal with the global renderer was to keep all the core backend logic centralized, so that we'd keep all the backend OpenGL calls in one place, rather than scattered across multiple classes

I kinda (mostly) agree with this. I think FlxGLRenderer could be the global way to interface with gl directly and call gl specific utils, where FlxGlView should be a render target who's public interface is defined in FlxCameraView. For example, in FlxGLView.hx:

var renderer(get, never):FlxGLRenderer;
inline function get_renderer() return cast (FlxG.renderer, FlxGLRenderer);

override function fill(color:FlxColor, blendAlpha:Bool)
{
    renderer.fillRect(getScreenRect(), color, blendAlpha);
}

This is somewhat pseudo code, I dunno if renderer.fillRect draws to the default buffer or whatever, but note that view.fill is defined in FlxCameraView, and implemented in this extension, however fillRect does not need to be defined in FlxRenderer, it can be defined and implemented entirely in FlxGLRenderer GL views have direct access to the gl renderer and can call gl specific things that other backends may not need to define. Likewise, anyone can assert that the current renderer is gl, if their project is set up that way, cast it and directly call gl methods on it.

That said, while all renderers don't need to define the same helpers/tools, we may want to define common tools in FlxG.renderer that people can access without caring about what backend is being used, but those global fields do not need to match view's common methods. Things like FlxG.renderer.drawMode = PIXEL_PERFECT

[ACrazyTown]

Yeah I think this could make sense. I don't know how I feel about implementing draw methods into FlxCameraView though, cause it might complicate other render targets unrelated to cameras. Perhaps it'd be better to implement them into a base FlxView class like you suggested?

I suppose what could really help clear things up is to actually try and implement the GL renderer into the new abstraction, which is what I'll be doing soon. Is your branch up to date? Not gonna merge here yet, but just so I know for reference locally

[Geokureli]

Gonna try a couple things, I'll let you know when I'm done, hopefully by sometime tomorrow

[Geokureli]

quoted from #3567

I'm a bit worried about tying the render methods to the camera view but I suppose this boils back to my previous sentence

The important distinction I wanna make is that, while we may move the rendering somewhere else, I do think it's important that to draw a sprite to a camera's view, you should call something akin to camera.view.drawThing(this) rather than something like FlxG.renderer.drawThingToView(camera.view, this), but viewBlit may end up calling privateRefToBlitRenderer.drawThing(this, sprite). I wanna focus on finding the best way to define these methods, before deciding exactly where they will end up.

FlxTexture, new methods that take bitmaps should take textures, instead

I have a draft (as discussed in #3540) for this typed up in a branch somewhere. It's implemented as a new class rather than an abstract so there is a bit of friction with implementing it with existing systems. Will need to revisit it and clean it up

The reason I make FlxVertexBuffer an abstract, is so that method args could be changed to take that type without breaking existing calls to it (as Graphics and FlxVertexBuffer can be implicitly casted to one another). Once devs migrate, FlxVertexBuffer will be a different type that will allow (or wrap) various underlying types. 1 option is to make it a class wrapper with derived types that use each specific underlying type. Another option is an abstract OneOfMany<T...> that calls the appropriate renderer

[ACrazyTown]

The important distinction I wanna make is that, while we may move the rendering somewhere else, I do think it's important that to draw a sprite to a camera's view, you should call something akin to camera.view.drawThing(this) rather than something like FlxG.renderer.drawThingToView(camera.view, this), but viewBlit may end up calling privateRefToBlitRenderer.drawThing(this, sprite). I wanna focus on finding the best way to define these methods, before deciding exactly where they will end up.

Yeah this makes sense. To be clear, I'm not against putting the render methods in camera view, I'm moreso against tying them to cameras specifically, as this would lead us further away from separating cameras and rendering (#1073).

It'll be easier to reason with this once we have all the missing pieces, so my focus now is to implement FlxTexture, FlxRenderTexture and the GL renderer (so that we can actually use the render texture). I already have basic quads working, so hopefully it won't take too long to get most things up and running.

In the meantime, I'd appreciate any help on #3566, as I think that's going to be one of the toughest parts to figure out.

The reason I make FlxVertexBuffer an abstract, is so that method args could be changed to take that type without breaking existing calls to it (as Graphics and FlxVertexBuffer can be implicitly casted to one another). Once devs migrate, FlxVertexBuffer will be a different type that will allow (or wrap) various underlying types. 1 option is to make it a class wrapper with derived types that use each specific underlying type. Another option is an abstract OneOfMany<T...> that calls the appropriate renderer

I don't think I can make FlxTexture an abstract because it introduces some new fields, but also works slightly differently. If devs need the actual BitmapData they can either:

• Use texture.handle to get the underlying representation. (Not recommended, if/when Flixel adopts other backends this will be changed to whatever type)
• or use texture.getBitmap(), which returns a abstract FlxBitmap(BitmapData) and also undumps the texture if needed (Recommended)

Hooking it up is doable, it's just an extreme hassle because everything expects a BitmapData. I'll try to finish and PR it here soon, tho