@ngeneva I have reworked the bent pipe example to something that should both have valid geometries, and can be usable as an example problem if nvidia is interested. However, like the previous problem I am having some difficulties getting it solve the water flow and would appreciate some assistance if you have the bandwidth.
This sample is emulating hot water flow in a T junction containing two inlets and one outlet. The pipe diameter is constant at ~5mm. The water relevant to my use case is
140f and so I’ve tried to select
rho values accordingly. This is not a space I’m experienced with however, and I may have selected these values poorly. I used this web app to aid me:
This example has been based on code from both the aneurysm and the 3D three fin samples.
velocity (m/s) = (0.5, 0, 0)
velocity (m/s) = (0, -0.5, 0)
pressure (Pa) = 0
Traditional CFD (expected) Solution
Scale is tied to velocity magnitude
Color is tied to pressure
t_junc.zip (56.8 MB)
||AMD Ryzen 3950x
In very early iterations (e.g. 1000-5000), Modulus produces output that appears close to the expected solution. This could just be a random coincidence given how few iterations have been ran, but it unfortunately fails to converge. As it continues to train, it will never solve the flow and continually diverge away from it. My current best guess is that this has something to do with scaling, as I’ve noticed that the modulus output produces velocities several magnitudes smaller than the CFD output. Unfortunately, I’m not sure how to troubleshoot that. I don’t believe it’s an over training issue because there’s no point in the middle at which it was producing good output either.
I’m hoping that someone can help me identify some fundamental flaw in my approach that’s preventing me from convergence. Thanks :-).
After 1000 training iterations
After 125k training iterations
Wow, thanks! I’ll do my best to take a look at this in the next week or two (will follow up). Really appreciate the effort here to set up a completely new system to share. Hopefully convergence can be improved as it would be an exciting edition to our examples.
Thanks @ngeneva, I’ll appreciate whatever you can offer.
Any chance you’ve had the opportunity to look into this @ngeneva ?
Means of facilitating PINN convergence is of interest to me as well.
I’ve bookmarked this problem, it may take me some time, but I will be examining other, related problems.
Apologies for the delay. I did take a look at the problem and did notice pretty slow training progress, however the overall problem set up appears good.
Regardless I have a few comments / suggestions I would eventually try on this type of problem which are below. Please correct me if something seems off:
- The first thing that I saw was that the integral continuity planes are positioned at the entrances of the pipe. For example this one here:
integral_continuity = IntegralBoundaryConstraint(
This is using the inlet mesh of the branch tube which results in an integral continuity plane just at the entrance of the mesh which isn’t actually that beneficial. Ideally we want these integral planes to be spaced out all along this inlet tube where we know the total flow rate. For example in the image below in paraview, the red points are the sampled current integral plane, the green are the volume and purple are where some good integral continuity could be placed.
Ideally you want these to be random and a few per batch. I think you could create a 2D circle primitive with a symbol for its Y location, one can then parameterize the Y variable in the integral continuity to sample circles along the vertical (Y) axis. (I think, havent tried so dont quote me)
Same deal for the other inlet tube (and even the outlet). Setting this up correctly will be likely to make a large difference in convergence.
What Reynolds number is this? I have not done the calculation myself, but if its quite high resolving the boundaries will be very challenging and I would probably recommend adding a zero equation turbulence model to the PDE (example). And also adjusting the sampling so you have a bulk flow constraint and then a near wall constraint. Can use the SDF as a criteria to accomplish this. We have used this in the past for some of the higher Re flows such as the FPGA problem.
I would have to look at the scaling more, but I’m thinking maybe theres some adjustments that could be made there to make the inlet velocities / viscosity with slightly more normalized values while maintaining the Reynolds number. Perhaps its alright as is.
Make sure you are not making the pipe diameter too small for the sake of decreasing the lengths of the tubes. If you know the inlet is likely laminar, you could decrease the length of the inlet pipes to make the diameter of the pipes larger since you know the flow profile (v(r) = v_0 (1 - r^2 / R^2) or something like that, cant recall exactly please check).
Consider using a fourier net model (keyword
fourier in the config for the model, docs are incorrect currently). They can work better for these problems with different length scales (in your case a long tube length but small diameter)
Unfortunately, I dont have bandwidth at this very moment to dig into this problem more. Although I do intend to keep it on my radar for the future. Maybe trying out some of these suggestions. Naturally there’s a number of things that could be tinkered with but the first priority is to get the flow to converge consistency (even at a slightly different Reynolds number although your validation will be useless).
Thanks for the feedback @ngeneva! I’ll definitely look into your suggestions. As for the integral planes, I don’t make the .stl files so to reduce how many new files I wait on I’m using the same mesh as the inlet/outlet, but then transforming it by an offset so that it’s somewhere along the length of the tube like you indicated. I believe the branch integral lands somewhere around the middle one you’ve marked. I hadn’t thought to parametrize the integral plane, but I can see how that’d be beneficial. Perhaps there’s a way I can parameterize the offset.
I did find a minor bug after I posted that code where
nu was getting multiplied by
scale twice, but resolving that didn’t get me much further.
I’ll have to follow up with more info about the Reynolds profile.
One of our goals after this sample converges is to parameterize the inlet velocity/volume flow, so I’m not sure messing with that would help me in the long run.
The pipe diameters are relatively fixed per the industry I work for, but there may be room to mess with changing the length.
I’ll be sure to update this thread if/when I get this working :)
For integral planes I would try to just use the CSG module! They are 2D circles so as long as you position it to be entered in the pipe it should be much easier than another STL file!
I’ll check that out then, that could be a much nicer solution :)
I tested the fourier network like you suggested and it made an immediate improvement. I recall testing all the different networks at one point, but perhaps there was another issue at the time that I’ve since resolved. At 15k iterations I can see the closest result I’ve seen so far to the truth. Pressure and scale are still pretty far off, but the fluid is flowing in a reasonable path now :D
One curious thing is that I’ve seen it’ll go from gradual progress to instantaneous insanity, which is coincidentally around the same iteration as specified under
decay_steps. This is the first bit of significant progress I’ve made, and it’s a good boost for my moral :). Wish me luck
Great news @npstrike ! Glad to see its making progress. : )
I’ve continued to chunk away at this as I’m able, and have made some progress. Per feedback about long tubes, I’ve tweaked the .stl’s to be shorter but still long enough to capture the information we’re looking for. I’ve been struggling to get this model to converge, and recently decided to test the assumption that it should converge at a rate similar to the Aneurysm sample which in my testing begins to show incomplete-but-good flow output between 15k-70k iterations. I have yet to see this T-Junction sample converge satisfactorily to a solution, but I’ve finally got it where it’s starting to look like convergence. The downside is that instead of taking thousands of iterations, it’s taking millions of iterations to show this progress. I’m reaching out to get some feedback, I’m beginning to think the setup may be ok and that the machine may eventually learn the flow profile, but it is taking too long.
Are there any suggestions for accelerating the model’s training?
- Inlet configured to a Reynolds number of ~1000
- Integral mass/volume flow planes set in the middle of both inlet branches and the outlet. These are about halfway between the inlet/outlet planes and the junction point in the center.
- Pressure on outlet plane set to 0
- Parabolic velocity has been disabled, and this model is currently running on a fixed inlet velocity
- SDF weighting is enabled
- Turbulence is not enabled, flow is expected to be laminar. I tried using ZE turbulence anyways, and it only confused the model.
I’ve done my best to provide context to the problem, but if I missed some important information, let me know. I’m also assuming that the amount of iterations I’m seeing is abnormally high. If this behavior is not unexpected, that’s also useful information.
I suspect I may be able to get benefits from a symmetry plane, and maybe exact continuity if I have the vram for it. I plan to experiment with those next. These may help reduce the time requirements to train and the pressure accuracy, but I don’t expect either to help reduce the number of iterations needed to converge.
Screenshot of the current state of training.
This screenshot highlights that the flow direction is consistent with what’s expected, and while the flow velocity is still wrong (outlet should have a velocity ~2x that of the two identical inlets), it is continually getting closer to the expected velocity magnitudes. Unfortunately, it is continuing to struggle with the pressure values compared to the expected pressures shown in the original post.
- Color shows pressure
- Size shows magnitude.
Screenshot of same model much earlier in the training process
This screenshot demonstrates the progress that’s been made in velocity magnitude and direction. Screenshot was taken around 150k iterations.
checkpoint_20230925.zip (53.1 KB)
Apologies, the source I linked had the wrong stl folder. Here’s a corrected zip.
stubbed.zip (65.1 KB)
I continued to allow this model to train, and it eventually converged on this as the solution:
This is by far the closest I’ve seen modulus come to producing the expected output. Unfortunately, it seems to have struggled with understanding the pressure in the top inlet, and never fully grasped the outlet’s velocity should be 2x the inlet velocity. My current hypothesis is that this took so long to train that the learning rate decreases eventually out-weighed it’s ability to learn.
Struggles with Exact Continuity
I attempted to switch the navier stokes eq.'s over to their exact-continuity counterpart seeing it listed as one of the recommended practices, however this only seemed to hurt my training. The loss values bounced around completely at random without any trend across ~1M iterations.
I added the following lines of code to my application, as that is what was recommended in the tutorials, but the lack of model improvement suggests that I’m missing something. Do you have any suggestions for me nvidia?
output_keys = [Key("a"), Key("b"), Key("c"), Key("p")]
c = Curl(("a", "b", "c"), ("u", "v", "w"))
nodes += c.make_nodes()
I’ll note that technically these lines don’t appear next to one another in the program so-as to fit my flow of logic, but it is in essence the only change I made for exact continuity.
If I can get this pipe to produce the same output as our CFD, then I can demonstrate to my company (A.O. Smith) that this technology can provide us value and continue pursuing our integration with it.