Shell-and-tube heat exchangers are among the most used thermal devices. Owing to their curved configuration, offer superior characteristics. Recently, nanofluids have gained significant attention due to their enhanced thermophysical properties. This study presents a comprehensive investigation on the influence of geometric and flow parameters on the thermal performance employing nanofluids. A three-dimensional computational model featuring a decreasing helical pitch in both horizontal and vertical directions was developed and compared with conventional constant-pitch configurations. Key thermal–hydraulic parameters, including the Nusselt number, pressure drop, heat transfer coefficient, and overall thermal efficiency, were analyzed using ANSYS Fluent. The findings demonstrate that the use of a decreasing helical pitch can enhance the Nusselt number and heat transfer coefficient by approximately 15–20%, while the associated pressure drop increases only by 5–8%. Furthermore, employing TiO₂–water nanofluid improved the overall thermal efficiency by up to 12% compared with the base fluid. These results confirm that integrating an optimized helical geometry with suitable nanofluids can substantially improve the heat transfer performance of shell-and-coil heat exchangers, offering valuable insights for energy-efficient industrial design.