AI-Driven Sales in Private Equity

2. Data Ingestion

Data Sources

The solution consolidated data from multiple sources:

• Salesforce Instances: Multiple environments containing customer relationship data.
• HubSpot Instances: Various sources for deal tracking and investment management.
• SQL Database: Structured data from transactional and operational internal systems.
• Preqin Feeds: Third-party data on private equity, venture capital, real estate and infrastructure fund performance, LP profiles, and fundraising activity.
• PitchBook Feeds: Market intelligence on company valuations, M&A and capital-raising deals, investor profiles, and fund metrics.

Challenges with the Existing Solution

Relying on a third-party package introduced steep costs, frequent failures, and lengthy issue resolution times:

• Excessive Subscription Fees: Ongoing high costs with little flexibility to scale down.
• Poor Data Design: Everything stored as nvarchar(max), causing performance bottlenecks and wasted storage.
• Unreliable Operation: The system threw over 10 errors per day, disrupting daily workflows.
• Heavy Customization Burden: Extensive tweaks were required to meet client needs, driving up implementation costs.
• Slow Support Turnaround: Every bug fix had to go through the vendor’s full ticketing process, delaying critical patches.

Our Solution

We rebuilt the existing 3rd party solution with custom PySpark code on Azure Databricks, landing raw feeds into Delta Lake and exposing them directly through Azure Synapse Analytics:

4. Data Quality Improvements

To address the firm's data quality challenges, a multi-layered data management framework was implemented, combining automated ETL pipelines with manual oversight:

• Format Standardization: Normalize common fields—trim whitespace, enforce consistent casing, apply ISO date formats, standardize phone-number patterns, strip non-UTF8 characters, etc.—to ensure downstream processes operate on clean inputs.
• Classification & Mapping: Map free-text or legacy codes into controlled vocabularies (e.g., industry segments, product categories, region codes) using lookup tables and rule-based logic, so that reporting and ML models work against a unified taxonomy.
• Deduplication:
  • Record Blocking & Indexing: Partition records into smaller “blocks” (e.g., by name initials or geographic hash) to limit pairwise comparisons and improve performance.
  • Fuzzy Matching & Phonetics: Compute string-similarity metrics (Levenshtein, Jaro-Winkler, n-gram overlap) and phonetic encodings (Soundex, Metaphone) on key fields.
  • ML-Driven Scoring: A supervised model ingests similarity features and transactional patterns to output a duplicate probability.
  • Thresholding & Workflow:
    • Auto-merge for pairs scoring ≥ 0.95
    • Pending-merge (0.75–0.95) for manual review via web app
    • Ignore for < 0.75
  • Continuous Learning: Human review labels feed back into model retraining to improve precision over time.
• End-User Data Collection & Validation: Request missing data from end-users and implement additional validation rules in their existing system to ensure completeness and accuracy.
• Audit Trail & Rollback: Log every change—automatic or manual—with before/after snapshots, user IDs, and timestamps, and provide rollback capabilities in case of mis-merges or mapping errors.

6. Custom Cross-Sell Model

Model Training

1. Feature engineering & selection
   Profiling and transforming raw data—normalizing financial metrics, encoding categorical attributes, and extracting sentiment signals from unstructured text.
2. Model experimentation & tuning
   Benchmarking multiple algorithms with extensive hyperparameter searches to balance speed and predictive power. Iteratively refining data quality and feature definitions whenever performance plateaued.
3. Validation & retraining
   Stress-testing each version on hold-out sets and pilot campaigns. Retraining several times to improve input data, recalibrate thresholds, and ensure business-ready robustness.

Auto-detected feature groups

• Financial Metrics: AUM, investment activity, fund performance
• Sentiment & Risk: investor sentiment scores, industry & country risk indexes
• Investor & Fund Attributes: investment horizon, vehicle type, strategy style, operational status

Performance Results

• ROC AUC: 0.78
• Precision Top 10 %: 0.32
• Recall (overall): 0.22
• F₁-Score: 0.26

Targeting the top decile of prospects delivers approximately a 3.5× lift over baseline response rates—empowering highly efficient, data-driven cross-sell campaigns.

Figure: Iterative training process for a custom machine learning model