The Limited
Victoria's Secret Pink

Case Study | Application Performance Engineering

About the Customer

Victoria’s Secret Pink is a wildly successful website, currently with over eleven million members. The creative company behind this viral digital marketing campaign was Resource/Ammirati. Resource is a brilliantly talented digital advertising agency, an exemplary corporate citizen, and our long term local partner.

The Challenge

The pink applications’ initial production performance baseline exhibited response time degradation at only seven concurrent users. Resource enlisted our team to provide 24×7 service desk, operations, technical support, hosting, and performance engineering of the entire Pink technology stack, as a turn-key solution. When we took over hosting the site, usage was growing exponentially, and mass email/text/social media advertisements were beginning to cause noticeable slowdowns.

Benefits

DCI developed a re-engineering plan that included graphical performance data from the grid monitoring system to compare each performance enhancement. DCI provided database and application performance tuning, network design and optimization, and extreme-scale cloud-based global performance testing. The test-analyze-fix cycles took about a month to increase performance from seven concurrent users to our target of forty thousand concurrent users.

Solution

Forensic Data Analysis

DCI’s data scientists processed all existing log data to analyze historic performance characteristics. The forensic data revealed that poor performance was more frequent and severe than qualitative observations suggested. The log data analysis also provided real-world usage patterns necessary to develop a test suite that accurately modeled common user workflow volume and relative frequency.

DevOps Automation

While our performance team established the initial performance baseline, our DevOps team completely automated the web application’s build, deployment, and testing processes. Automating the build improved the efficiency and reliability of releases, but more importantly, it gave us the opportunity to inject custom instrumentation code into the codebase. The instrumentation provided detailed metrics for all of the application’s external dependencies, providing new insight into the real-time status of virtual machines, cache providers, resource pools, databases, and external service providers. The metrics were available as a secure web service that only impacted performance when explicitly invoked.

Grid Monitoring

Once the performance team had established a repeatable initial performance baseline, our infrastructure team instrumented the operating systems, virtual runtime environments, database servers, application servers, web servers, and network devices. The always-on low overhead monitors provided hundreds of metrics that were aggregated and displayed graphically using an open-source distributed grid monitoring dashboard. An open source alerting tool interrogated the grid monitor to notify stakeholders of any resource constraints and track availability incidents.

Performance Engineering

Our engineers used empirical data gathered from the stack instrumentation during repeated load tests to identify and tune each inefficient configuration item, independently measuring and documenting the relative performance of every change. The instrumentation easily identified a major bottleneck in the database connection pool configuration. Other database optimizations were necessary at over 100 transactions per second, including explain plans that yielded inefficient/missing indexes, pessimistic concurrency causing unnecessary blocking operations, a hyper-threading incompatibility that caused cache misses, etc. We meticulously documented the relative performance increase of each change that we made to the system. The database tier accounted for the majority of the significant performance opportunities. As database performance improved, the web and application server tiers benefited from minor heap, thread, and directive optimization. Operating system, load-balancer, firewall, and network tuning accounted for negligible performance improvement relative to application data concurrency engineering.

Results

Victoria’s Secret Pink is a wildly successful website, currently with over eleven million members. The creative company behind this viral digital marketing campaign was Resource/Ammirati. Resource is a brilliantly talented digital advertising agency, an exemplary corporate citizen, and our long term local partner.